Interface and application #

Intro #

In this page, I will document the process of building an interactive web interface for the ESP32 board.

Idea #

The idea was to combine three things together via the wifi connection that was covered in the Networking week:

• An interactive new media art named Squishy Stack based on p5.js
• The ESP32 board
• The MPU-6050 gyro/accelerometer

Here’s a playable version of the original Squishy Stack by Ivan Rudnicki.

You can also try it on OpenProcessing.

I was immediately fascinated by the ‘juicy’ feel it provided after I played for a while, the way of interacting with the stack, squishing and squashing them feels so satisfying, almost like playing with a blob of jelly with your hand. I was also surprised to see that this exquisite visual effect was achieved with just a few lines of code, the simplicity of the implementation is truly remarkable.

So naturally, I thought this unique interactive experience would be even better if we can have a physical object and use that to engage embodied interaction. The gyroscope/accelerometer was the best choice for this idea, so I went on to test it.

Arduino #

I build the Arduino server with the similar structure as the Networking week. The main differences are:

• Used async web server commands to handle request from the browser.

// Serve the JSON payload
server.on("/json", HTTP_GET, [](AsyncWebServerRequest *request){
  sendJsonResponse(request);
});

• Used json to format and transfer sensor data.

DynamicJsonDocument doc(1024);
doc["angleX"] = ax;
doc["angleY"] = ay;
doc["angleZ"] = az;
doc["accX"] = acx;
doc["accY"] = acy;
doc["accZ"] = acz;
String jsonString;
serializeJson(doc, jsonString);

• Added access control in the header.

AsyncWebServerResponse *response = request->beginResponse(200, "application/json", jsonString);
response->addHeader("Access-Control-Allow-Origin", "*");
response->addHeader("Access-Control-Allow-Methods","*");
response->addHeader("Cross-Origin-Resource-Policy", "cross-origin");
request->send(response);

Therefore, the main setup for the arduino server looks like this:

void setup(){
  // Serve the JSON payload
  server.on("/json", HTTP_GET, [](AsyncWebServerRequest *request){
    sendJsonResponse(request);
  });
}

void sendJsonResponse(AsyncWebServerRequest *request) {
  Serial.println("Sending Json respond");
  Serial.print(ax);Serial.print(",");
  Serial.print(ay);Serial.print(",");
  Serial.print(az);Serial.println();

  DynamicJsonDocument doc(1024);
  doc["angleX"] = ax;
  doc["angleY"] = ay;
  doc["angleZ"] = az;
  doc["accX"] = acx;
  doc["accY"] = acy;
  doc["accZ"] = acz;
  String jsonString;
  serializeJson(doc, jsonString);
  Serial.println(jsonString);
  AsyncWebServerResponse *response = request->beginResponse(200, "application/json", jsonString);
  response->addHeader("Access-Control-Allow-Origin", "*");
  response->addHeader("Access-Control-Allow-Methods","*");
  response->addHeader("Cross-Origin-Resource-Policy", "cross-origin");
  request->send(response);
}

Also I deleted all the OLED displaying code, since they tend to interrupt the server and sensor thread for some reason.

#include <Adafruit_SSD1327.h>
#include "Wire.h"
#include <MPU6050_light.h>
//#include "math.h"
#include <ArduinoJson.h>
#include <WiFi.h>
#include <ESPAsyncWebSrv.h>
//include <AsyncJson.h>

// Settings for WIFI connection
const char* ssid     = "Fablab";
const char* password = "Fabricationlab1";   

// Setting up webserver
AsyncWebServer server(80);
//This function will be called when human will try to access undefined endpoint
void notFound(AsyncWebServerRequest *request) {
  request->send(404, "text/plain", "Not found");
}



// Settings for MPU6050 Accelero/Gyro
MPU6050 mpu(Wire);
unsigned long timer = 0;

// Angle data from gyro
float ax = 0;
float ay = 0;
float az = 0;
float acx = 0;
float acy = 0;
float acz = 0;

// Coordinate and parameters for drawing
int x = 0;
int y = 0;
const int radius = 10;
const int t_size = 10;
float rad1 = 0;
float rad2 = 0;
float rad3 = 0;



// Settings for OLED
#define OLED_CLK SCK
#define OLED_MOSI MOSI
#define OLED_CS D2
#define OLED_DC D3
#define OLED_RESET D0
Adafruit_SSD1327 display(128, 128, OLED_MOSI, OLED_CLK, OLED_DC, OLED_RESET, OLED_CS);



// Settings for JSON
//StaticJsonDocument<200> jsonDocument;
//String jsonString;



void setup() {

  Serial.begin(9600);
  Wire.begin();
  // WIFI start
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");

  // We want to know the IP address so we can send commands from our computer to the device
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());


  // Serve the JSON payload
  server.on("/json", HTTP_GET, [](AsyncWebServerRequest *request){
    sendJsonResponse(request);
  });

  //OLED start
  Serial.println("SSD1327 OLED test");
  
  if ( ! display.begin(0x3D) ) {
     Serial.println("Unable to initialize OLED");
     while (1) yield();
  }
  display.clearDisplay();
  display.display();

  display.setTextSize(1);
  display.setTextColor(SSD1327_WHITE);
  

  //MPU start
  byte status = mpu.begin();
  Serial.print(F("MPU6050 status: "));
  //display.println("MPU6050 status: ");
  Serial.println(status);
  //display.println(status);
  while(status!=0){ } // stop everything if could not connect to MPU6050
  Serial.println(F("Calculating offsets, do not move MPU6050"));
  //display.println("Calculating offsets, do not move MPU6050");
  //display.display();
  delay(1000);
  // mpu.upsideDownMounting = true; // uncomment this line if the MPU6050 is mounted upside-down
  mpu.calcOffsets(); // gyro and accelero
  Serial.println("Done!\n");
  //display.clearDisplay();
  //display.display();
  server.begin();
}



void loop() {
  mpu.update();
  ax = mpu.getAngleX();
  ay = mpu.getAngleY();
  az = mpu.getAngleZ();

  acx = mpu.getAccX();
  acy = mpu.getAccY();
  acz = mpu.getAccZ();

  // display.clearDisplay();
  // display.setCursor(0,0);
  // display.print("angleX:");display.println(ax);
  // display.print("angleY:");display.println(ay);
  // display.print("angleZ:");display.println(az);

  // display.print("accX:");display.println(acx);
  // display.print("accY:");display.println(acy);
  // display.print("accZ:");display.println(acz);
  // display.display();
  //Serial.print(ax);Serial.print(",");
  //Serial.print(ay);Serial.print(",");
  //Serial.print(az);Serial.println();


  /*
  // Calculate coordinate
  x = map(long(ax),90,-90,0,display.width());
  y = map(long(ay),-90,90,0,display.height());

  // Draw circle
  display.drawCircle(x, y, radius, SSD1327_WHITE);

  // Draw triangle
  rad1 = az / 180 * M_PI;
  rad2 = (az+120) / 180 * M_PI;
  rad3 = (az+240) / 180 * M_PI;
  display.drawTriangle(int(x + sin(rad1)*t_size), int(y + cos(rad1)*t_size),  
                  int(x + sin(rad2)*t_size), int(y + cos(rad2)*t_size),  
                  int(x + sin(rad3)*t_size), int(y + cos(rad3)*t_size), SSD1327_WHITE);
  display.display();
  */
}

void sendJsonResponse(AsyncWebServerRequest *request) {
  Serial.println("Sending Json respond");
  Serial.print(ax);Serial.print(",");
  Serial.print(ay);Serial.print(",");
  Serial.print(az);Serial.println();

  DynamicJsonDocument doc(1024);
  doc["angleX"] = ax;
  doc["angleY"] = ay;
  doc["angleZ"] = az;
  doc["accX"] = acx;
  doc["accY"] = acy;
  doc["accZ"] = acz;
  String jsonString;
  serializeJson(doc, jsonString);
  Serial.println(jsonString);
  AsyncWebServerResponse *response = request->beginResponse(200, "application/json", jsonString);
  response->addHeader("Access-Control-Allow-Origin", "*");
  response->addHeader("Access-Control-Allow-Methods","*");
  response->addHeader("Cross-Origin-Resource-Policy", "cross-origin");
  request->send(response);
}

p5.js #

Given that the gyroscope/accelerometer sensor provides only general orientation information, I needed to make certain modifications to the original code to accommodate this simpler input data. So now only the swaying effects got preserved.

for (let v of this.verts) {
  v.r += v.rv;
  v.rv -= (v.r - r) / 10;
    
  // if (movedX!=0 || movedY!=0 ) {
  // let d = dist(v.pos.x, v.pos.y, mouseX - width / 2, mouseY - height / 2) / 2;
  // v.rv -= r / max((height / 50), d);
  // }
  v.rv *= 0.9;
  v.pos.x = v.r * cos(v.a);
  v.pos.y = v.basey + (v.r / this.h) * sin(v.a);
  curveVertex(v.pos.x, v.pos.y);
}

Then I added the code that gets data from the Arduino server.

// Get Gyro data from the Xiao board
function getData(){
    fetch('http://193.167.5.162/json')
    .then(response => {
        if (!response.ok) {
        throw new Error('HTTP error, status = ' + response.status);
        }
        return response.text();
    })
    .then(textData => {
        // Transform the text into JSON
        let jsonData = JSON.parse(textData);

        // Access the JSON data
        ax = jsonData.angleX;
        ay = jsonData.angleY;
        az = jsonData.angleZ;

        acx = jsonData.accX;
        acx = jsonData.accY;
        acx = jsonData.accZ;
        // Print the data to the console
        //console.log('ax:', ax);
        //console.log('ay:', ay);
        //console.log('az:', az);
    })
    .catch(error => {
        console.error('Error:', error);
    });
}

The fetch function did not work in the p5.js web editor, but it worked in local environment, which costed me a lot of time for debugging. My guess is that the p5.js web editor have stricter license control, which denys fetch request from our illegal server even after added the access control header.

Results #

After struggling with javascript for hours, I finally managed to get all this work together. Here’s the result that I got.

As you can see in the video, the response was laggy, which is frustrating that it completely ruins the experience. But it’s still fun to try it out, and I learned a lot about javascript along this process.